Dynamic foveal vision display

ABSTRACT

A head mounted display system with at least one retinal display unit having a curved reflector positioned in front of one eye or both eyes of a wearer. The unit includes a first set of three modulated visible-light lasers co-aligned and adapted to provide a laser beam with selectable color and a first scanner unit providing both horizontal and vertical scanning of the laser beam across a portion of the curved reflector in directions so as to produce a reflection of the color laser beam through the pupil of the eye onto a portion of the retina large enough to encompass the fovea. The unit also includes a second set three modulated visible-light lasers plus an infrared laser, all lasers being co-aligned and adapted to provide a color and infrared peripheral view laser beam, and a second scanner unit providing both horizontal and vertical scanning of the visible light and infrared laser beams across a portion of the curved reflector in directions so as to produce a reflection of the scanned color and infrared laser beams through the pupil of the eye onto a portion of retina corresponding to a field of view of at least 30 degrees×30 degrees.

FIELD OF THE INVENTION

The present invention relates to head-mounted display systems and inparticular to foveated head-mounted vision display systems.

BACKGROUND OF THE INVENTION Head Mounted Displays

The most rapid transfer of information to humans is through vision. Headmounted displays are a modality of human-computer interface associatedwith vision. Head mounted display devices are well known. They are asmall display device worn on the head sometimes as a part of a helmet.They may also be mounted on or be a part of a visor, goggles oreyeglasses. Head mounted displays can operate in either of two modes. In“augmented reality” mode the display is see-through, and the displayimagery is superimposed upon natural vision. In “virtual reality” mode,the display is occluding and blocks the view of the local scenery,entirely replacing it with displayed imagery. The performance of currenthead mounted displays is limited compared to the typical human visualcapability. Current head mounted display devices have serious ergonomicissues that significantly handicap the wearer. Examples of prior artdesigns for head mounted display devices includes a device with a goggleformat proposed by William Schonlau and described in a SPIE paper,“Personal Viewer; a wide field, low profile, see-through eyeweardisplay”, SPIE Vol. 5443, 2004 and “Immersive Viewing Engine”, SPIE Vol.6224, 2006. This device is retinal scanning display based head mounteddisplay with a curved primary mirror in front of the eye. FIG. 17A is ablock diagram showing an overview of Schonlau's design and FIG. 17B isconceptual overview of the proposed prior art scan process.

Virtual Retinal Displays

Some head mounted displays are based on a technology referred to as“virtual retinal display” or “retinal scanning display”. This is atechnology that draws a raster display (like a television) directly ontothe retina of the eye. The users see what appears to be a conventionaldisplay floating in space in front of them. This technology was inventedby Kazuo Yoshinaka of Nippon Electric Company in 1986. Later work at theHuman Interface Technology Laboratory at the University of Washingtonprovided much smaller devices and these devices have been developed andmarketed by Microvision, Inc. with headquarters in Redmond, Wash. FIG.1A is a sketch showing a typical prior art virtual retinal display. Itincludes a video graphics array input, controlling electronics, one ormore light sources, which are typically lasers, a light modulator,vertical and horizontal scanners and delivery optics. Light weightscanning systems are available utilizing two-dimensional MEMS scannerswith at least one axis mechanically resonant. FIG. 1B shows twoschematic views of such a device illustrating the two axes of motion.These devices are a few millimeters in size and replace the two largerscanning mirrors shown in FIG. 1A. In addition, it is now possible todirectly modulate diode lasers with the required bandwidth, and anexternal laser modulator is not required. Laser-based scanning engineswith direct diode laser modulation, and using 2D MEMS scanners areutilized in many modern display systems and are commercially availableoff-the-shelf. Some of these displays have laser outputs that aredesigned to be directed directly into the eye and are focused by theeye's optics directly onto the retina of the eye.

Prior Art Eye Tracking

Prior art methods of eye tracking are expensive, and insufficientlyreliable. These prior art methods of eye-tracking typically image thepupil of the eye using an infrared light source and an additional camerato monitor infrared light reflected from the retina. Images from theextra camera are also used to locate glints from one or more infraredlight emitting diodes. The best prior art desktop eye tracking systemstypically have a reliability of between 85% and 90%. The 10%-15% ratesof error become very annoying to users of gaze contingent displays. Theprocessing power required to search for glints, determine the center ofthe pupil, and then calculate the gaze direction can be large andincompatible with the requirements for a portable head mounted display.Glints from objects illuminated by sunlight can interfere with theglints from the infrared light emitting diodes.

Limitations of Current Head Mounted Displays

Conventional head mounted displays have several serious limitations.Foremost is the tradeoff between spatial resolution and field-of-view.Conventional head mounted displays typically have both a limited fieldof view (20°-30°) and limited spatial resolution. The available pixelsfrom some form of flat panel device must in general be spread uniformlyover the desired field of view. The resolution is limited due to thefinite pixel count and bandwidth restrictions. A very large pixel-countdisplay (such as about 4200 pixels per dimension) would be required toprovide both high-resolution (20/20 equivalent vision) and a field ofview of 70°. Such a display does not to Applicants' knowledge exist inthe prior art. The bandwidth required to deliver the required pixelinformation would present another set of difficulties involving size,weight and power consumption. Displays with even wider fields of vieware desirable.

Another limitation of conventional head mounted displays is that theyare typically heavy and bulky, often consisting of an optical systemthat might be described as a “stack of glass pancakes” especially forwide fields of view. Regardless of the display field of view provided,typically the see-through field of view is impeded. This is true, evenif only a flat fold-mirror is placed in front of the eye. Inconventional displays, see-through transmission is typicallysignificantly impacted through the use of broadband partial reflectors,or holographic elements. For military applications, light leakage withpartial reflectors or light scattering with holographic elements impactcovertness, especially at night.

Yet another limitation of conventional head mounted displays is thatthey cannot properly display 3D imagery. In order to properly display 3Dimagery, retinal disparity must exist appropriate to the depth of eachobject in the imagery, and in addition the focus of the object mustagree with the depth. When these two depth cues fail to agree, thewearer will often suffer from what is known as simulator sickness.Typical head mounted displays have no provision for controlling thedisplay focus or for making the display focus agree with real imagery inaugmented reality mode.

SUMMARY OF THE INVENTION

The present invention overcomes the above limitations, and providesadditional advantages over conventional head mounted displays. Thepresent invention provides a virtual retinal display that is consistentwith natural eye function. It provides high-resolution vision over alarge field of view combined with low bandwidth requirements. The headmounted portion of the present invention is lightweight, compact andergonomic. It can properly display 3D imagery.

Preferred embodiments of the present invention provide a head mounteddisplay system with at least one retinal display unit having a curvedreflector positioned in front of one eye or both eyes of a wearer. Theunit includes a first set of three modulated visible-light lasersco-aligned and adapted to provide a laser beam with selectable color anda first scanner unit providing both horizontal and vertical scanning ofthe laser beam across a portion of the curved reflector in directions soas to produce a reflection of the color laser beam through the pupil ofthe eye onto a portion of the retina large enough to encompass thefovea. The unit also includes a second set three modulated visible-lightlasers plus an infrared laser, all lasers being co-aligned and adaptedto provide a color and infrared peripheral view laser beam, and a secondscanner unit providing both horizontal and vertical scanning of thevisible light and infrared laser beams across a portion of the curvedreflector in directions so as to produce a reflection of the scannedcolor and infrared laser beams through the pupil of the eye onto aportion of retina corresponding to a field of view of at least 30degrees×30 degrees. An infrared light detector detects infrared lightreflected from the retina and subsequently the curved reflector andproduces an infrared reflection signal which is used by controlelectronics to determine the current view direction of the eye. The unitalso includes a video graphics input device providing color videographics input signals. Based on the view direction of the eye and thevideo graphics input signals, the control electronics modulates theintensity of the components of both beams as they scan the curvedreflector so as to direct the foveal beam onto a portion of the retinaencompassing the fovea and to direct the peripheral view beam onto aportion of the retina corresponding to a field of view of at least 30degrees×30 degrees.

As explained in the Background section, achieving both high resolutionand a wide field of view simultaneously is very difficult in headmounted displays. If this was achieved by simply filling a large pixelcount flat-panel array (e.g. liquid crystal displays or organic lightemitting diode arrays) with high-resolution imagery, the computationalpower required to drive the display would be largely wasted becausehigh-resolution is useful to the eye only in the foveal region where itis currently looking, and everywhere else only a reduced resolution isrequired. Applicants break up the displayed imagery into ahigh-resolution foveal component and reduced resolution peripheralcomponent, and the wearer's eye obtains high-resolution imagery onlywhere it is needed. In preferred embodiments, Applicants use anunconventional, robust and lightweight eye tracker to determine the gazeangle direction and generate a high resolution 20/20 visual zonecovering the approximately 10° diameter field of view associated withthe fovea (which Applicants refer to as the “foveal field of view”centered on where the user is currently looking), with a total field ofview of 70° or more. The resolution is appropriately reduced in theperipheral vision. This greatly reduces the bandwidth requirements.Preferred embodiments are rugged, low-cost and provide high-resolutionimagery.

Due to the use of retinal scanning display technology, the “pixels” ofimagery are generated by laser beams entering the eye from an array ofangles. The divergence of the laser beams as they enter the eyespecifies the focus of the imagery. In a preferred embodiment a focusadjuster is incorporated to vary the divergence of the scanning laserbeams, thereby controlling the focus of the imagery.

The key innovations of present invention include curved primary mirrorsin front of the eyes that are lightweight and provide a large displayfield of view, and in augmented reality mode, a large unimpededlook-through field of view. Preferred embodiments utilize narrowbandrugate reflective coatings in augmented reality mode to maximizetransmission of light from the outside scenery, yet ensure that nosignificant amount of light can escape from the display. These preferredembodiments permit, for example, a soldier, pilot or surgeon to viewlive environments in augmented reality mode with a substantiallyunimpeded field of view and unimpeded transmission. The presentinvention includes an eye tracker that is simple and robust. Preferredembodiments utilize existing retinal scanning display hardware.Preferred embodiments use direct diode laser modulation andtwo-dimensional MEMS scanners to provide laser beams which are reflectedfrom a curved mirror into the eye of the wearer so as to provide highresolution only where it is needed, and low resolution everywhere else.Preferred embodiments use a reverse “wavefront coding” concept tocorrect residual aberrations resulting from the use of the curvedmirror. In one preferred embodiment, the curved mirror in front of theeye is spherical and the display field of view is 70°. In anotherpreferred embodiment the mirror in front of the eye is ellipsoidalallowing for an extremely large display field of view such as 120degrees. The eye is in some sense a megapixel device as the optic nervecontains approximately 1 million fibers. The MEMs-based laser scanningengines proposed for the present invention each have 0.92 megapixelcapability. Since one scanner is utilized for foveal vision and one forperipheral vision, the information capacity of the proposed head mounteddisplay rivals what the eye is capable of receiving.

BRIEF DESCTIPTION OF THE DRAWINGS

FIG. 1A is drawing describing features of a prior art retinal scanningsystem.

FIG. 1B is a drawing showing a prior art MEMS scanner.

FIGS. 2A, 2B and 2C show features of a preferred embodiment of thepresent invention.

FIGS. 3A-3D illustrate how wavefront coding is utilized for thecorrection of optical system aberrations.

FIG. 4 is from a prior art paper and is a plot of the modulationtransfer function versus spatial frequency of an optical system with andwithout wavefront coding and with and without an added defocusaberration.

FIGS. 5, 6A, 6B, 6C, and 6D are prior art drawings showing features ofexit pupil expanders.

FIG. 7 depicts the purpose of an exit pupil expander in increasing thesize of the eye box.

FIGS. 8A and 8B depict the basic mechanism of the eye-tracking whichutilizes retro-reflection from the eye.

FIGS. 9A and 9B depict a system view of the eye-tracking based uponretro-reflection from the eye.

FIG. 10 shows an important feature of the present invention thatprovides an increased pixel density and thus higher display resolutionin the foveal portion of a wearer's field of view.

FIG. 11 demonstrates how applicants utilize an exit pupil expander toassure that at least one laser beam enters the wearer's pupil.

FIG. 12 is a prior art chart showing the time required to change focusfrom near to far as a function of age.

FIG. 13 demonstrates one method of determining the correct focus of thedisplay, by determining how far away the two eyes are pointing

FIGS. 14, 15A, 15B and 15C demonstrate the effects of various focalstates of the on retro-reflected light.

FIGS. 16A and 16B show a technique for adjusting the focus of thedisplay to match actual view distances.

FIGS. 17A and 17B show features of a prior art retinal scanning system.

DETAILED DESCRIIPTION OF PREFERED EMBODIMENTS

Applicants' technology provides an ideal solution to the problems ofprior art head mounted displays. Pixels are generated wherever andwhenever they are needed in the field of view through judiciousmodulation of the lasers. The lasers are operated at high bandwidth overthe foveal field of view where the wearer is currently looking, andoperated at appropriately reduced bandwidth in the peripheral field ofview. In this manner the wearer perceives high-resolution imagerywherever they look over a large field of view, yet the requiredbandwidth is manageable. Preferred embodiments can be described byreference to FIGS. 2A through 11. Applicants use two separate twodimensional MEMS scanner systems (each with a horizontal and verticalscan motion similar to the scanner shown in FIG. 1B). These scanners areavailable from suppliers such as Microvision, Inc. with offices inRedmond, Wash. One of the scanner systems produces the fovealhigh-resolution image, and a second scanning system produces the widefield of view peripheral image. A curved polycarbonate primary mirror infront of the eye is ergonomic, compact and lightweight. In preferredembodiments adapted for augmented reality modes the mirror reflects thelaser light but also provides minimal reduction of see-through field ofview in wrap around geometries and simultaneously provides maximumimpact resistance. The display doubles as a form of eye protection.Diode lasers generate light with higher wall plug efficiency than anyother technology, so the light generation is energy efficient. Optimumchoice of the wavelengths utilized results in a color gamut that isimpossible to obtain in any other manner and greatly exceeds thatavailable from conventional displays. Brightness of the display in fullsunlight is not a problem. The use of a dynamic focus adjuster for thelaser beams allows for proper rendition of three-dimensional imagery.The use of narrowband rugate coatings for the primary mirror results inunsurpassed see-through transmission and negligible light leakage. If aflat fold mirror were utilized in front of the eye, then the laser beamswould need to be collimated when viewing at infinity, and any leakedlight would also be collimated. Instead, the use of a curved mirrorensures that any leakage of display light will be divergent (notcollimated) and rapidly dissipate and become undetectable, ensuringcovertness for applications requiring covert operation.

The use of a curved mirror requires a method to correct for the opticalaberrations inherent in its use. The laser supported retinal scanningtechnology dovetails with the use of a curved mirror in that the higheffective f-number of the laser beams dramatically reduces theaberrations induced by the curved mirror. In fact, without the laserscanning technology, the use of a curved primary mirror is probably notfeasible. Even with the laser scanning, there are residual aberrationsthat could adversely affect resolution at larger field angles. To removethese residual aberrations in preferred embodiments, Applicants utilizea novel wavefront coding technology to correct in software for theaberrations before the images are displayed. Applicants' foveationtechnique requires a robust form of eye-tracking. Conventionaleye-trackers are expensive, bulky and encounter problems functioning inbright sunlight due to interference. Worse, they are not as robust as isdesired. Gaze-contingent displays become very annoying when theeye-tracking fails. As shown in FIG. 2B, Applicants add to theperipheral field of view scanner an infrared laser 10 in addition to thecolored laser stack 12 of red, green and blue lasers. An avalanchephotodiode 14 also incorporated into the laser housing records and mapsthe “red-eye” retro-reflection from the eye. The retro-reflectionpattern is essentially a cone of 20° full-width half-maximum in angularspace centered on the gaze direction, with some small constant offsetthat is determined during a calibration step. Every time the displayimage updates, the current gaze direction is robustly determined, andthe high-resolution portion of the displayed image is moved to becoincident with the gaze direction. Very little extra hardware isrequired, permitting the device to remain compact and lightweight.Because the gaze direction is determined using the scanning displaysystem beam path, there is no chance of losing registration.

Basic Architecture of the Present Invention

Preferred embodiments of the present invention are based on directwriting to the retina of the user's eye, using laser beams and MEMSscanners. This approach in conjunction with the curved polycarbonateprimary mirror permits a reduction in size, weight and powerrequirements for a head mounted display while achieving high resolution(20/20 vision) and a large field of view (50°×70°). A 10° diameterfoveal high-resolution zone is generated in the gaze direction where theuser is currently looking. A lower-resolution peripheral image isgenerated in the remainder of the field of view. FIG. 2C is a blockdiagram showing the principal elements of a prototype system designed byApplicants. It includes a high definition video input 20 and a foveationprocessor 22 for determining the portion of the video input to displayusing the foveal scanner 30. The system also includes a first set ofco-aligned red, green and blue lasers 12 to provide full color imagesand an eye tracking infrared laser 14 (also co-aligned with the visiblelasers) furnishing a light beam for the wide field scanner 24 via afirst aperture sharing element (ASE) 25. The gaze direction is monitoredby an unconventional robust lightweight eye-tracker including anavalanche photodiode (APD) 26 and an eye-tracking processor 28. Theeye-tracking processor 28 controls a foveal scanner 30 which receivesillumination from a second set of co-aligned red, green and blue lasers32 which are controlled by wavefront coding processor 34. The wavefrontcoding processor is described in detail below. The two scanned beamsfrom the wide field scanner 24 and the foveal scanner 30 are combined ina second aperture sharing element 36 and directed to display optics 38where the combined beam is directed into the user's eye. Applicants'system when designed for an augmented reality application has thefollowing advantages as compared to conventional head mounted displays:full color, wide display field of view and high resolution, occluded ornon-occluded modes, unimpeded natural field of view, see-throughtransmission exceeds 80%, high brightness adequate for operation in fullsunlight, tolerant to slippage, lightweight (25 grams/eye) for thehead-mounted portion, low power (a few watts), negligible leakage to theoutside environment and eye protection from impact and external lasersources. Applicants' prototype version is mounted directly on the facesimilar to a pair of aviation or safety goggles. A schematic and overallarchitecture are shown in FIG. 2A-C.

Laser Displays are Safe

Some lasers are potentially dangerous to the eye only because they arebright sources of light, meaning that they are capable of being focusedto a small spot on the retina, dramatically increasing the power perunit area or flux. Damage can result if the flux delivered to the retinais too high. However, any source of light can cause damage if theoptical flux produced at the retina is too large. In an imaging system,higher resolution is directly associated with the capability of thesystem to produce smaller spot size on the retina. It is frequently thegoal of the designer of a display system to produce the smallestpossible spot size. Lasers are ideal for imaging applications becausethey are particularly good at being focused to small spot diameters.Safety depends upon controlling the flux delivered to the retina, as forany other source of light. There are two categories of commercialproducts that direct a laser beam into the eye. In the first categorydirection of the laser beam into the eye is intentional and in thesecond category the direction is not intentional but is neverthelesssafe. The first category includes wavefront aberrometers, retinalscanners and laser scanning ophthalmoscopes. The second categoryincludes supermarket checkout scanners and laser radar for measuringautomobile speed. In all of these commercial products, safety is ensuredby controlling and limiting the maximum flux delivered to any spot onthe retina. Typical retinal display engines utilize laser powers on theorder of hundreds of nano-watts and are unconditionally eye safe.Preferred embodiments of the display described herein include a safetysub-system that turns off the lasers in case of a scanner fault and/or amodulation fault. Other embodiments utilize lasers which can be safelydirected into the eye such as ANSI Class 1 lasers, so that even if thereis a scanner or modulation fault, and even if in addition the safetysub-systems fail, the eye is at absolutely no risk of damage.

Reverse Wavefront Coding

Virtual retinal displays using curved primary mirrors were previouslythought impossible due to the difficulty in controlling the aberrationsintroduced by the curved mirror. However the advantages of a curvedprimary mirror are considerable, including a compact and lightweightdesign, potentially immense display field of view, and a substantiallyunimpeded look-through field of view. Virtual retinal display technologyutilized by Applicants dramatically reduces the aberrations induced bythe curved mirror by reducing the area of the mirror used to generateeach pixel (to approximately a beam diameter).

Reverse “wavefront coding” is a technology that can eliminatesubstantially all of the remaining aberrations and provide for highresolution at all field angles. The point spread function describes theresponse of an imaging system to a point source of light, such as astar. Aberrations in an optical system such as the optical system shownin FIG. 2C (without the wavefront coding) in general distort the imageby creating a larger and/or less symmetrical point spread function. Thebasic process of wavefront coding involves two steps:

-   -   (1) First Applicants place a phase plate in the optical system        that essentially determines a constant system point spread        function despite additional aberrations from the optical system        itself, which vary with field angle. That phase plate        aberrations exist with this property comprises the essential        “magic” of wavefront coding techniques. Typically, a wave plate        with a third-order aberration involving coma and trefoil is        used, although other aberrations types have also been found to        work. Defocus, astigmatism and spherical aberration added by the        optical system will have negligible impact on the point spread        function in the presence of the third-order phase plate        aberration provided that their collective amounts are less than        the amount of third-order phase plate aberration as measured by        root mean square wave front error. Figuratively speaking, the        third-order aberrations dominate the profile of the point spread        function.    -   (2) The second process is to in software de-convolute from the        images the known and constant point spread function due to the        wave plate prior to display. Upon subsequent display, the        de-convoluted image is re-convoluted in hardware with the        third-order aberrations. The net effect is to restore the images        to near pristine quality removing the effects of the optical        system aberrations.

One penalty for utilizing such a technique is that there is added powerconsumption to perform the required de-convolution image processing.However, modern image processing chips are compact, energy efficient,powerful and affordable. This is an example of a hybridoptical-electronic system.

Traditionally wavefront coding is used to increase depth of focus of animaging device such as a microscope or cell phone camera. In theseapplications the image is de-convoluted in software after capture withan imaging system that includes a third-order wave plate. In the currentscheme, the image is de-convoluted in software prior to display usinghardware that includes the third-order wave plate. The Applicants referto this as “reverse” wavefront coding.

As indicated above wavefront coding preferably requires that the phaseplate aberrations are larger than or equal to the aberrations due to theoptical system. If this is not the case, then the point spread functionwill begin to be affected by the hardware aberrations, and the techniquewill have reduced effectiveness. The known phase plate aberrations areeliminated by de-convolution in software. The typical wavefront codingtradeoff is that some excess signal-to-noise ratio is consumed in theprocess. In the particular case of the present invention, the pointspread function determined by the phase plate is de-convoluted from theimages prior to their display. The display will then, in hardware,convolute the de-convoluted image with the known point spread functionproducing a much improved image for the eye. In this manner thedifficult and varying aberrations due to the use of a curved mirror infront of the eye are circumvented. High resolution is possible for allfield angles. Applicants have performed simulations of wavefront codingto correct for various combinations of added aberrations.

The basic process of wavefront coding is described in FIGS. 3A, B, C andD and FIG. 4. FIGS. 3A-3D and FIG. 4 are from a SPIE paper: “WavefrontCoding: A modern method of achieving high performance and/or low costimaging systems”, E. Dowski and G. Johnson, SPIE Vol 3779 (1999) pages137-145. FIG. 3A depicts the point spread function of an idealdiffraction-limited optical system. FIG. 3C shows the point spreadfunction of a diffraction-limited system in the presence of thethird-order aberrations intentionally added for purposes of wavefrontcoding. FIG. 3B shows the point spread function for an optical system inthe presence of a significant amount of defocus aberration. Of note hereis that the smearing due to the defocus is isotropic and severe. FIG. 3Dshows the point spread function of an optical system containing both thedefocus of FIG. 3B and the third-order aberration of FIG. 3C. Oneimportant point is that the severe isotropic smearing implied in FIG. 3Bhas been removed. A second important point is that there is nodetectable difference between the point spread functions shown in FIG.3C and that shown in FIG. 3D. Therefore, if the point spread functionshown in FIG. 3C is de-convoluted from the optical system of FIG. 3D,not only will the effects of the third-order aberration be removed, butalso the effects of the defocus shown in FIG. 3B will be removed aswell. Hence the effects of varying amounts of aberrations can be removedwith the technique of wavefront coding. The modulation transfer functionof an optical system is a measure of the quality of an image created bythe system. FIG. 4 is a graph showing the results of the use ofwavefront coding to correct aberrations such as those resulting from theuse of curved mirrors in virtual retinal displays. There are three setsof two curves. Each set of two shows the modulation transfer functionwith and without an added defocus aberration. The thick curves are themodulation transfer function of an optical system without added defocus(upper curve) and with added defocus (lower curve). Adding defocusdepresses the modulation transfer function, destroying the quality ofany image made with the optical system. At certain spatial frequencies,the depression is almost to the baseline meaning that there is little orno recoverable information at that spatial frequency. The lower two thincurves (one with added defocus, the other without added defocus) are themodulation transfer function of the same optical system with an addedthird-order aberration for purposes of wavefront coding. As can be seen,the addition of the defocus does not further degrade the modulationtransfer function in the presence of the third-order aberration. Inaddition, the MTF curve in the presence of the added defocus is notdriven to the baseline at certain spatial frequencies, as for the casewith added defocus alone, and the image information is recoverable. Theupper two thin curves are the after de-convolution of the addedthird-order aberration. There is little difference between the curvewith added defocus, and the curve without the defocus. Hence thewavefront coded system is immune to the effects of the amount of addeddefocus. Furthermore, the modulation transfer function of the wavefrontcoded system actually exceeds that of the base optical system withoutwavefront coding at many spatial frequencies.

The processing power requirements and associated power consumption tohandle the needed calculations have been investigated and Applicantshave determined that currently available processors can provide therequired computations and corrections in real time using compact andportable electronic processors. Ophthonix, Inc with offices in Vista,Calif. has developed a technology to manufacture highly accurate phaseplates containing arbitrary phase patterns, as required for wavefrontcoding. Use of an electrically addressable spatial light modulator toprovide the wavefront coding phase pattern would enable one to turn offthe wavefront coding so as to conserve the processing power when highresolution imagery is not needed. In addition, such an electricallyaddressable spatial light modulator could double as the variable focuselement.

Exit Pupil Expansion

All head mounted displays illuminate an area of the face known as the“eye box” located at the exit pupil of the display system asschematically shown in FIG. 7. If the eye is in the eye box, thedisplayed image will be visible. If the eye is outside the eye box, thedisplayed image will not be visible. It is desirable to have the eye boxlarge enough that the head mounted display can be easily adjusted toplace the eye in the eye box. It is also desirable to have the eyeremain inside the eye box despite relative movements of the display andface due to anticipated activities. One advantage of the face-mountedgoggle version of the present invention is that relative motion betweenthe head mounted display and the face is anticipated to be relativelysmall despite rigorous activity of the wearer. Therefore the size of theeye box can be considerably smaller as compared to the helmet-mountedversion. A smaller eye box means that less light is wasted illuminatingthe face adjacent to the eye, and simultaneously the amount of possiblelight leakage that could theoretically compromise covertness in militaryapplications is also reduced.

FIGS. 5, 6A, B and C, 7 and 11 schematically illustrate the concept ofexit pupil expander (EPE) 40 utilized to expand the size of the eye boxat the exit pupil. FIGS. 5, 6A, 6B and 6C are from a SPIE paper:“Microlens array-based exit pupil expander for full color displayapplications”, H. Urey and K. Powell, Proceedings of the SPIE volume5456, April 2004, and earlier publications referenced therein. Thedevice takes a relatively narrow beam from the scanner, and incombination with the final imaging optic, creates numerous parallelbeams to increase the size of the eye box. Details of the function areshown in FIGS. 6A, B, C and D. This existing technology simply requiresa design with parameters appropriate to the new proposed device.Preferred embodiments of the present device utilize an exit pupilexpander as shown at 40 in FIGS. 5 and 11. The prior art MicrovisionMicro-Lens-Array Exit-Pupil-Expander (MLA-EPE) is a refractivediffractive hybrid optic that expands the exit pupil for all wavelengthsequally. The exit pupil expander may be flat or curved. A curved exitpupil expander, as shown in FIG. 11, would reduce the field-dependantaberration components significantly. However, curved is significantlymore difficult to fabricate; the stronger the curvature the moredifficult the fabrication. A flat exit pupil expander would be easy tofabricate but creates larger field-dependent aberrations. An exit pupilexpander is made up of two precisely aligned lenslet arrays as shownschematically in FIG. 6A.

The pupil of a human eye has a diameter that varies from 2 mm to 8 mmbut is typically about 3 mm in conditions of good illumination. The exitpupil expander basically converts one input beam into a large array ofparallel beams that are spaced apart into a hexagonal array. It isimportant that at least one beam of the array of beams enter the pupil.In the exit pupil expander shown in FIGS. 6A, B and C (which is designedfor a helmet mounted display), a single narrow beam is converted into 91parallel beams distributed over an aperture roughly 15 mm in diameter.The distance between adjacent beams is of the order of 1.5 mm. Thisassures that at least one but more typically several beams will enter a3 mm pupil with each pulse of light from the stacked lasers providedthat the pupil is located somewhere in the pattern of beams. For thegoggle mounted display the pattern of spots would preferably be muchsmaller. Preferred patterns would include seven or fourteen parallelbeams as shown at 41 and 42 in FIG. 6D.

To begin the fabrication task, each lenslet array begins with a maskthat is used to etch the lenslet shape into a master tool. The mastertool is then used to replicate the lenslets in epoxy onto appropriatelyshaped substrates. Each lenslet array generally has its own pitch anddepth and therefore requires its own mask and master. Once fabricated,the two lenslet arrays are precisely aligned to each other and bonded.For the flat exit pupil expander, field flattening lenses will befabricated and bonded to the exterior of the exit pupil expander toappropriately accommodate the incoming and outgoing beams. Thefabrication of the curved exit pupil expander requires a few additionalsteps if the mask and master are flat. The epoxy lenslet array is madeon a flat surrogate substrate and is much thicker. It is then removedfrom this substrate and placed on a curved substrate.

Novel Eye Tracking Scheme

A portion of light incident upon the eye is retro-reflected.Retro-reflection is a generic feature of all imaging systems that employa detector at the focus of a lens. For the human eye that detector isthe retina. Photographer's red-eye reflection is an example. A techniqueutilized in photography is that in order to avoid the red eyeretro-reflection, the flash and camera should be separated by at least 3degrees, because the retro-reflection from the eye is fairly narrow.Optical calculations using a simple eye model indicate that theefficiency of retro-reflection varies with incident field angle fairlydramatically due to light trapping behind the iris as shown in FIG. 8A.A calculation of this effect has been presented in an unpublished paperby C. E. Mungan at the U. S. Naval Academy entitled “The Cat's EyeRetroreflector” available athttp://www.usna.edu/Users/physics/mungan/Scholarship/scholarship.html.Consequently, if one probes the eye with beams of light from alldirections (FIG. 8B), the amount of retro-reflection will peak in thegaze direction and will fall off fairly dramatically away from the gazedirection. In order to produce an image in focus at infinity, thepresent invention generates beams of collimated light directed towardthe eye from a dense array of angles filling the display field of view.The beams are produced by red, green and blue lasers that areappropriately modulated. With the addition of an infrared laser that isnot visible and an avalanche photo-diode to look for retro-reflectedinfrared light, the retro-reflection magnitude is mapped over the fieldof view to determine the angular location of peak retro-reflection,which is the gaze direction (with some constant offset determined in acalibration step). Thus the gaze direction will be determined at thedisplay frame rate, which is at least 60 frames per second. FIGS. 9A and9B depict the eye tracker system concept. Advantages of this schemeinclude the fact that little extra hardware is required, and thetracking is quite robust. In addition, since the gaze direction isdetermined using the display hardware, there are no registration errors.Robustness in ensured because the conical pattern of retro-reflectioncentered on the gaze direction is broad in angle and encompasses asubstantial proportion of the display pixels. The amount of redundantdata specifying the conical pattern is very large. Errors on a fewpixels, whatever the cause, cannot alter the overall pattern.

Foveation to Achieve High Resolution

High resolution in a display with a wide field of view is achieved bycreating a foveated display. A high resolution image is displayedcovering a roving 10°×10° zone centered on the current gaze direction,and an image with reduced resolution is displayed over the remainder ofthe field of view. A resolution of 1 arc-minute (equal to 1/60 of onedegree) is required for 20/20 vision (the 20/20 “E” is 5 arc minutestall). Therefore the 10°×10° foveal zone requires of the order of600×600 pixels to provide 20/20 caliber resolution. This is similar tothe number of pixels in a video graphics array (VGA) (640×480) device ora wide video graphics array (WVGA) (848×480) device. The number offibers in the human optic nerve has been counted, and the answer isslightly over 1 million fibers (F S Mikelberg et al., The normal humanoptic nerve: Axon count and axon diameter distribution, Ophthalmology,96(9) 1989, pp 1325-8). In this sense, the entire human eye (foveal plusperipheral vision) is roughly a megapixel device. If the 10°×10° fovealzone utilizes 600×600=3.6×10⁵ pixels, then the peripheral vision wouldaccount for the remainder or 6.4×10⁵ pixels. These are rough estimates.The acuity of the young adult eye is actually better than 20/20 (i.e.about 20/13.5 on average). In addition the peripheral eye performs localcalculations before transmitting data to the brain. However, the abovenumbers act as a guide to what might be useful for a display to providein terms of information content.

Resolution in a laser scanning display is controlled by two factors.Vertical resolution is controlled by the angular density of horizontalscan lines. Horizontal resolution is controlled by the minimum laserpulse duration in combination with the angular scan speed. To avoid thephenomena of flicker, the frame rate should be at least 60 per second.Therefore the frame rate cannot be reduced arbitrarily to increaseresolution.

If an available scanner has sufficient resolution to generate thedesired foveal resolution anywhere in the field of view when the pixelsare uniformly distributed over the entire field of view of the display,then only one such scanner is required per eye. In this case, resolutionand average bandwidth are controlled by modulation of the laser beams.The laser beams are modulated at a high rate for a high pixel count inthe foveal zone, and modulated at reduced rate for a much lower pixelcount in the remainder of the field of view. In this manner, the averagenumber of effective pixels is modest despite the generation of highresolution imagery everywhere the wearer looks over a large field ofview. Retinal scanning display technology is steadily improving, andretinal scanners with very large effective pixel counts are expected tobe available in the next few years. Microvision's current wide videographics arrays scanners offers 480×848 pixels. Scanners providingadditional pixels are expected in the near future.

For 20/20 caliber foveal vision and a total field of view comprising50°×70°, the required effective pixel count from a single scanner is3000×4200. Current high definition (HD) TV displays are only 1080×1920pixels so the requirement represents an advanced technology. To achievefoveation in this case, the Applicants utilize two scanner systemscombined using a beam splitter. One scanner generates the peripheralvision scene and may also be used to generate a portion of the fovealscene resolution. The other scanner generates the foveal scene or theremainder of the foveal scene if the task is shared with the peripheralscanner. The foveal scanner is operated in a novel fashion.

Microvision prior art scanners utilize resonant scanning in thehorizontal direction to conserve power. The vertical direction is drivenby a ramping voltage applied to the other axis of the MEMS mirror. Inpreferred embodiments the horizontal axis always makes full scansresonantly as it is designed for. The vertical ramp, however, is notover the full field of view, but instead only over a vertical range of10° centered on the current vertical position of the foveal gaze, asshown in FIG. 10. The vertical position of the foveal zone is adjustedby moving the scanned zone up and down within the total field of view tomatch the current gaze angle. The horizontal position of the foveal zoneis adjusted via timing of the laser pulses so that the laser pulses areproduced only when the scanner is directing pulses to the foveal zone.By scanning over 10° instead of the full vertical field of view, thedensity of horizontal lines is increased proportionally, therebyincreasing the resolution. If the foveal scanner lines are interlacedappropriately with the peripheral scanner lines in the foveal region,the resolution can be further increased. When the foveal imagery isproduced only with the foveal scanner, the peripheral scanner may beinstructed to display nothing in the foveal zone.

In cases where the total display field of view is very large, it may benoted that it is not necessarily desirable to provide foveal resolutioncapability over the entire peripheral field of view. This is because thehuman eye typically does not look more than ±20° from straight ahead. Infact it is difficult to do so. Instead people turn their heads to lookat objects more than 20° from a straight ahead gaze. Therefore, if forinstance a display offers a total horizontal field of view of 120°(±60°), one may only need to supply foveal resolution over the center50° (±25°) field of view. The horizontal field of view beyond ±25° isthen always dedicated to peripheral vision. In this manner thehorizontal foveal resolution can be increased over what is possible ifthe foveal scanner horizontal scan lines had to instead cover the entirefield of view.

Rugate Coatings for Augmented Reality

In the augmented reality or see-through mode of the display, it isdesirable that the primary “mirror” in front of the eye have excellenttransmission yet reflect the laser beams generating the overlaidimagery. Rugate coatings are optical surface coatings in which the indexof refraction of the applied layers is made to vary in a continuousfashion. Their advantage is in the creation of reflectors with very highoptical density but extremely narrow bandwidth. Hence, a substrate witha rugate coatings may appear crystal clear and have 90% transmission,yet totally reflect laser beams at several chosen wavelengths. Rugatecoatings are therefore ideal for an augmented reality retinal scanningdisplay. Three-color rugate coatings have already been deposited oncurved substrates for use with light emitting diode driven heads-updisplays and have demonstrated 80 percent see-through transmission. Thecurrent application utilizes lasers with narrower bandwidth, whichpermits designs with greater see-through transmission, and even greateroptical density at the reflected wavelengths.

In a preferred embodiment, the primary mirror in front of the eye iscomprised of polycarbonate plastic. Polycarbonate plastic hasunsurpassed impact resistance and is utilized in almost all safetyglasses for this reason. Fortunately, polycarbonate is a standardoptical plastic widely used in the ophthalmic industry. Machining andpolishing of polycarbonate to optical tolerances is widely available.Anti-reflection coatings and hard-coats are readily available andinexpensive.

Focus Adjuster

There are several good reasons for incorporating a focus adjuster inhead mounted displays. In augmented reality, the display overlay shouldbe in focus with the background image, so that both can be visualizedsimultaneously. A display that only focuses at infinity will be oflittle use when viewing closer objects and will prove annoying in thosesituations. In both augmented reality (see-through mode) and virtualreality (occluded mode) visual clues relating to depth must agree toprevent the nausea often referred to as simulator sickness. Typically toobtain three dimensional images, the retinal disparity is provided, butnot the correct focus corresponding to the vergence. In cases of largemotion in depth, a significant number of individuals will eventuallyexperience nausea when only retinal disparity is provided to indicaterange. It has been proposed that this is due to an evolutionaryadaptation in which the brain decides that the only way such conflictingsignals can arrive at the brain is if a dangerous substance has beenconsumed. Consequently an urge to throw up (nausea) is generated. Thesolution to all of the above issues is to include a focus adjuster inthe display so that the display overlay is in focus with the backgroundobjects being viewed (augmented reality mode) and so that vergence andfocus depth cues agree (both modes).

In augmented reality mode, the display overlay should be in focus withthe current object being viewed. The simplest implementation is to havethe entire display at the same focus, which may change with time as thewearer focuses on different objects in the field of view. What isrequired is a method of determining the focus of the eye, so that thedisplay can match it.

In virtual reality mode, the object currently being viewed with fovealvision should be displayed with a focus appropriate to its depth. Otherobjects need not have a focus appropriate to their depth since they arenot currently being looked at with foveal vision. Hence the displaydevice could simply provide a constant focus for the entire currentimage that is appropriate to the object in the image currently beinggazed at. If one did not know which pixel corresponds to the center ofthe visual field, the defocus of all pixels would have to be correctedin real time. This would be practically impossible due to the enormousbandwidth required for the focus adjuster. However, the Applicant's headmounted display includes eye tracking to achieve foveation. As such itwill be known where the wearer is looking and the focus can be adjustedso that the primary object being looked at has a focus appropriate toits depth. The focus adjuster need only keep up with the accommodationof the eye. In the Applicants head mounted display, an adjustment ofseveral diopters could utilize up to half second and still keep up withthe fastest accommodating eyes. The proposed focusing technology is,however, much faster than the requirement.

LensVector, Inc. based in Mountain View, Calif. has developed and ismarketing an electronically addressable variable lens for use inproducts such as cell phones. The base technology involves liquidcrystals and optical power change is induced with changes in voltage.The liquid crystal layers are thin, and two such layers are utilized,one for each orthogonal polarization. The external transmission is 90%.Their standard lens is 4.5 mm×4.5 mm×0.5 mm and weighs 22 mg. The drivercan be reduced to 2.1 mm×1.4 mm×0.2 mm. The driver utilizes only cellphone voltages. The driver automatically compensates for variations incomponents and environmental conditions. The lens requires less thanhalf the power of mechanical alternatives in cameras. There are nomacroscopic moving parts. Only the liquid crystal molecules move, so thedevice is silent. The standard lens is designed to vary focus frominfinity to 10 cm, a range of 10 diopters. This range of focus is morethan adequate for the head mounted display application. Unpowered thedevice is essentially a sheet of glass and has no optical power. Thetransition is faster in one direction than the other. The time to changefocus 1-diopter is of the order of a few tens of milliseconds in onedirection and a few milliseconds in the other direction. The entire10-diopter range can be scanned on the order of a second. The standardaperture is actually larger than required in the head mounted displayapplication, and smaller apertures can change focus faster than largerapertures. The LensVector variable lens is being mass produced for usein cell phones, and as such is a relatively low cost component. Anotherpotential small, low-cost focus adjuster solution is the adjustablefocus lenses of the type described in U.S. Pat. Nos. 7,232,217 and7,338,159 (which are incorporated herein by reference) awarded toSpivey. These lenses each includes two lens elements having specialsurfaces such that an adjustment of the position of one of the twolenses relative to the other in a direction perpendicular to the viewingdirection will produce a change in focus.

In the Applicant's head mounted display, the focus adjuster has to befast enough to keep up with the eye. Accommodation changes are actuallyquite slow compared to video rates, and therefore the defocus adjusteris not required to have a high bandwidth. It just has to be compact andpower efficient. How fast can a person accommodate? The following datais from “The Time Required for U.S. Navy fighter Pilots to Shift Gazeand Identify Near and Far Targets”, Ailene Morris and Leonard Temme,Aviation, Space and Environmental medicine, Vol 60, (November 1989) pp.1085-1089. In this study, subjects were required to recognize theorientation of a Landolt C optotype at 20/20 resolution, both at 18inches and then at 18 feet. The minimum time for the pair of optotypesto be correctly recognized in succession was measured. A plot of theresults is shown in FIG. 12. The minimum average time exceeds 500 msecfor the youngest and fastest accommodating subjects. Other studies suchas “Age, degraded viewing environments, and the speed of accommodation”,Charles Elworth, Clarence Larry, Frederick Malmstrom, Aviation, Spaceand Environmental Medicine, Jan 1986, pp. 54-58 have measured muchlonger times for accommodation. By comparison, the pixel dwell times forWVGA at 60 Hz update are of the order of 15 nsec. If the focus adjusterhad to operate at the pixel time scale, the task would nearly beimpossible. In the Applicant's head mounted display, adjustment ofseveral diopters can utilize a half second and still keep up with thefastest accommodating eyes.

Methods of Auto-Refraction

In augmented reality mode, the focus of the overlay display should matchthe current focus of the wearer. Therefore methods are required todetermine the current focus of the wearer, or in optometrist language,refract the wearer. The Applicants propose two different methods toaccomplish this.

Method #1

In the first method, the convergence of the two eyes is measured. Byexamining the pointing of the two eyes, it is possible to determine howfar away they are looking and to then set the focus appropriately. Inbinocular mode with dual eye-trackers, the convergence can be directlycalculated and the defocus adjusted accordingly. In other words, usingthe proposed eye tracking, the gaze angle of each eye will be known.Therefore the distance of the object being viewed can be calculatedusing trigonometry. The necessary defocus can then also be calculatedand subsequently implemented in the display. Referring to FIG. 13, therelationship between the gaze angles and the distance to the objectbeing viewed for the simple case of looking straight ahead is given by:

$\begin{matrix}{{\tan (\theta)} = \frac{D}{2L}} & (0.1)\end{matrix}$

Hence if the uncertainty in the gaze angle θ is given by δθ, theuncertainty in the vergence (1/L) is given by:

$\begin{matrix}{{\delta \left( \frac{1}{L} \right)} \approx {\frac{2}{D} \cdot \frac{\delta \; \theta}{\cos^{2}(\theta)}}} & (0.2)\end{matrix}$

The inter-ocular distance D is approximately 6.5 cm, so for most objectdistances L, the following approximation can be made:

$\begin{matrix}{{\cdot {\cos (\theta)}} = {\frac{L}{\sqrt{L^{2} + \left( {D/2} \right)^{2}}} \approx 1}} & (0.3)\end{matrix}$

Hence the uncertainty in vergence is given by:

$\begin{matrix}{{\delta \left( \frac{1}{L} \right)} \approx {{\frac{2}{D} \cdot \delta}\; \theta}} & (0.4)\end{matrix}$

If L and D are given in meters, then the uncertainty in vergence δ(1/L)is given in diopters. For an inter-ocular distance D equal to 0.065 mand for δθ equal to half a degree (typical conventional eye trackingaccuracy), the uncertainty in vergence is approximately 0.25 diopters.For δθ equal to 0.1° (roughly the best conventional eye tracking) theuncertainty in vergence is about 0.05 diopters. Spectacles are typicallyprescribed with 0.25-diopter accuracy. With good eye tracking accuracy,the uncertainty in vergence is negligible.

Method #2

The second method of determining the focus of the wearer is to vary thedefocus adjuster performing a search so as to maximize the eye-trackingretro-reflection signal. In this manner the eye tracker couldsimultaneously auto-refract the eye, maximize the eye-tracking signaland automatically focus the display at the plane of the objectscurrently being viewed.

FIG. 14 schematically depicts the situation when the eye is focused atinfinity, and collimated light is incident upon the eye. The collimatedlight is focused to a point on the retina. Any light reflected from theretina is retro-reflected due the reversibility of the geometric lightpaths. FIGS. 15A, 15B and 15C schematically illustrates what happenswhen collimated light is incident upon the eye, but the eye is focusednot at infinity, but instead upon a closer object as shown in FIG. 15A.In this case the collimated beam is defocused upon the retina andilluminates a larger region as shown in FIG. 15B. Reflected light fromeach illuminated spot on the retina produces an optical path such asthose illustrated in FIG. 15C. Most of these paths do not constitute aretro-reflection. The consequence of this is that the retro-reflectedlight collection efficiency is significantly diminished.

If a focus adjuster were included in the system, the retro-reflectedlight signal would be maximized when the eye-tracking probe light wasincident upon the eye with the same divergence as from a point on theobject being viewed. Hence, by varying the focus adjuster to maintainmaximum retro-reflection signal level, the associated display would bekept in focus with the real objects that the eyes are currently viewing.

FIGS. 16A and 16B illustrate a system view. In FIG. 16A the eye islooking through the primary mirror and is focused upon a point on a realobject closer than infinity. Consequently the light from this objectthat enters the eye is more divergent than it would be if the objectwere a considerable distance away. When the display illumination isgiven the identical divergence as light from the external point beingviewed then both the external point and the display light will besimultaneously in focus on the retina. The colored display illuminationand infrared probe light have identical divergence because they sharethe same optical path and are both affected equally by the focusadjuster. In practice, the divergence of the output of the infraredlaser is slightly adjusted by using a lens at its output so as tocorrect for chromatic aberration in the focus adjuster lens. As depictedin FIG. 16B, when the colored display illumination and the infraredprobe light are matched to the current focus of the eye, the infraredprobe light is efficiently and optimally retro-reflected back down theoptical path and the signal at the avalanche photo diode is maximized.Therefore, by adjusting the focus adjuster so as to maximize theavalanche photodiode signal and thus maximize the retro-reflectionefficiency, the visible display illumination is automatically forced tobe in the same focus as light from whatever external object the eye iscurrently focused upon. The focus adjuster focusing power is variedperiodically to search for and maintain the maximum retro-reflectionsignal through the use of an algorithm such as a gradient ascentalgorithm.

Electronics Implementation

Applicants have determined that required software and processing powercan be implemented using a field programmable gate array (FPGA) andeventually an application specific integrated circuit (ASIC). Thesedevices are compact, lightweight and have power requirements compatiblewith portability.

Head Tracking

To provide maximum utility in both augmented reality and virtual realitymodes, the displayed image should have the capability to move inresponse to head motion. For instance, the wearer in virtual realitymode could turn their head to view new portions of the surroundingscene. A wearer in augmented reality mode could turn their head andobtain augmented reality overlays appropriate to other objects in theirsurrounding environment. This is possible with display orientationtracking. Using a MEMS gyro sensor such as that available in the WiiMotion Plus devices, such tracking may be possible at a reasonable cost.Certainly for military training or gaming in an occluded mode, thiswould be a significant and useful advance in hardware capability. Foraugmented reality, the displayed image must move with head motion tomaintain correspondence with the real world. Image processing algorithmswill be required to shift the overall scene to correspond to the currentdisplay and head orientation. In an embodiment of the Applicant'sdevice, MEMS gyros are incorporated into the head-mounted display sothat head orientation tracking can be implemented.

Size, Weight and Power Requirements

Preferred embodiments of the present invention can be produced in both amonocular and a binocular version. A version useful for deployment inthe military would probably have to fit over corrective eyewear,although a spectacle prescription could be implemented directly in thedisplay lenses themselves.

The anticipated size and weight of the head mounted component of theproposed device is similar to that of a pair of safety goggles. Thedevice described in William Schonlau and referred to in the Backgroundsection of this specification has such a format. As indicated thatdevice was described in “Personal Viewer; a wide field, low profile,see-through eyewear display”, SPIE Vol. 5443, 2004 and “ImmersiveViewing Engine”, SPIE Vol. 6224, 2006. This device is retinal scanningdisplay based head mounted display with a curved primary mirror in frontof the eye. However, the author did not have an adequate solution to theproblem of aberrations due to the curved mirror, did not implement anyscheme to achieve foveation, he did not present a scheme for eye tackingrequired for foveation, and he did not present a scheme for varying thefocus of the display.

Preferred embodiments could include a second electronics component thatcould be connected to the head mounted component either wirelessly orwith a wire. The second component could be a belt-mounted unit. It couldalso be incorporated into the console of a television set or gamingconsole. The size and weight of the belt mounted unit are expected to besimilar but slightly larger than a pico laser projection display unit,which incorporates image processing electronics and a battery powersupply in addition to a laser scanning projector with power adequate forprojection on a wall. Dramatically less laser power is required for thehead mounted display. The pico display unit weighs 122 grams and hasoverall dimensions of 14 mm×60 mm×118 mm.

Estimates of power consumption depend upon the level of development ofthe product. Demonstration prototypes consume significantly more powerbecause the processing is not implemented in energy efficient ASICchips. Applicants estimate power consumption for the preferredembodiment described in FIG. 2C as follows:

-   -   Foveal scanner prototype 1.0 W, product 0.6 W    -   Peripheral scanner prototype 1.8 W, product 0.8 W    -   Wavefront coding processing 3 W with standalone NVIDIA, 450 mW        with FPGA/ASIC    -   Eye-tracking processing 3 W with standalone NVIDIA, 300 mW with        FPGA/ASIC    -   Foveation processing 2 W with standalone NVIDIA, 100 mW with        FPGA/ASIC    -   Eye-tracking laser & APD 5 mW

The total estimate is a peak power consumption of 11 W per eye for thedemonstration prototype, and 2.3 W per eye for a product with theelectronic processing implemented in ASICs.

Many tricks may be possible to further reduce power consumption. Forinstance, the scanners need operate only when there is information to bedisplayed. Due to the eye tracking, it will always be known where theeye is looking. Most of the time the wearer will typically look throughthe center portion of the primary mirror in front of the eye whereaberrations are low and aberration correction may not be required. Ifthe wavefront coding was turned on only when it was required, powersavings would be significant. This would require a removable phase plateor an electronically programmable phase plate. Such devices exist andare being developed. Another idea is to consider the resolutionrequirements of a displayed image. If only lower resolution is required,then the wavefront coding resolution enhancement may temporarily not berequired. In fact the foveation itself could be temporarily turned offif the required display resolution is low.

Prototype System

Thus, a prototype system designed by Applicants provides the followingattributes:

-   -   Exceptionally high resolution where the wearer is looking (20/20        vision in one preferred embodiment)    -   Appropriately reduced resolution for peripheral vision to        utilize a manageable bandwidth    -   Wide display field of view (such as 50°×70°) in one preferred        embodiment)    -   Wide, unimpeded see-through field of view with excellent        transmission in augmented reality mode    -   Extraordinary color rendition    -   Sufficient brightness for use in bright sunlight in augmented        reality mode    -   Undetectable light leakage for covertness in augmented reality        mode    -   Protects the eyes by providing unsurpassed impact resistance    -   Minimal size, weight, and power requirements    -   Can provide full 3D imagery with correct focus cues in addition        to retinal disparity cues    -   In augmented reality mode automatically focuses the displayed        imagery at the same depth as objects currently being viewed by        the wearer

Subsidiary Advantages of the Present Invention

The proposed head mounted display offers a number of subsidiaryadvantages, some of which are listed in the following:

Laser eye protection—In augmented reality mode, the rugate coatings onthe primary mirror in front of the eyes also provide laser eyeprotection for the wavelengths utilized in the display. The rugate stackcould be augmented to include other wavelengths known to present adanger.

Verification of identity—The eye-tracking scanner records the structureof the retina on top of the overall retro-reflection pattern, much as ascanning laser ophthalmoscope would do, and can be used to verifyidentity.

Detection of certain medical conditions—The eye-tracking data could beused to monitor a number of health related issues. For instance, byexamining the retro-reflection levels of the various colors used in thedisplay, the oxygen saturation could be easily determined and monitored.

Detection of fatigue or incapacitation—When a person is about to fallasleep, the pupil of the eye begins to fluctuate in size. Thismodulation will be recorded by the eye-tracking software and could beused to alert the user or command structure. If a pilot were to blackout due to g-forces, the eye tracking system would detect the signatureand the plane could be instructed to go to autopilot, saving both theplane and the pilot.

Communication via eye movements or blinking—The eye tracking systemcould be utilized as a secondary communication system. For instance, ifthe user was injured but conscious, they could blink their eyes in someprearranged code to inform the command system via the eye trackingsystem. The wearer could operate machinery in this manner.

Determination of alertness and effort—The harder people think, thelarger their pupils become. If searching for a target, when the targetis located, the pupils will momentarily enlarge. There is a lot ofinformation that can be ascertained by observing the wearer's eyes withthe eye tracking system.

Variations

Although the present invention has been described in terms of certainpreferred embodiments, persons skilled in the art of head mounteddisplays will recognize that there are many changes and variations thatcould be applied within the general concepts of the invention. Forexample: the curved primary mirror in front of the eye could beimplemented on a helmet mounted visor. The curved primary mirror infront of the eye need not be directly head mounted. The curved primarymirror in front of the eye could be implemented on an aircraft cockpitwindow. The curved primary mirror in front of the eye could beimplemented on an automobile window. The retinal scanning technologydoes not require lasers. Super luminescent diodes (SLDs) and lightemitting diodes (LEDs) could be utilized in place of lasers, forexample. Three visible lasers are only required for full color displays.A single laser is adequate for a monochrome display. More than threevisible lasers can be utilized to further increase the color gamutpossible, providing more colors than most people would have a chance tosee in any other manner. All aspects described in this document need notbe simultaneously implemented. For instance, the curved mirror in frontof the eye may be utilized with wavefront coding for aberrationcorrection. However, foveation and the associated eye tracking may notbe implemented if the resolution and field of view do not require it.

Therefore the reader should determine the scope of the invention by theappended claims and not the specific examples that have been given.

1. A head mounted display system comprising at least one retinal displayunit said at least one display unit comprising: A) a curved reflectorand a frame adapted to position the curved reflector in front of atleast one eye of a wearer, said at least one eye defining a pupil, aretina, a fovea and a view direction, B) a first set of at least threevisible light lasers all lasers being co-aligned and adapted to providea co-aligned, color foveal laser beam, C) a second set of at least threevisible light lasers plus an infrared laser all lasers being co-alignedand adapted to provide a co-aligned, color and infrared retinal laserbeam, D) a first scanner unit adapted to provide both horizontal andvertical scanning of the co-aligned color laser beam across a portion ofthe curved reflector in directions so as to produce reflections of thecolor laser beam through the pupil of the eye onto a small portion ofthe retina, said small portion being less than 20 percent of the retinabut large enough to encompass the fovea, said small portion defining afoveal region, E) a second scanner unit adapted to provide bothhorizontal and vertical scanning of the co-aligned color and infraredlaser beam across a portion of said curved reflector in directions so asto produce a reflection of the scanned color and infrared laser beamthrough the pupil of the eye onto a portion of retina corresponding to afield of view of at least 30 degrees×30 degrees, F) an infrared lightdetector adapted to detect infrared light reflected from the retina andthe curved reflector and produce an infrared reflection signal, G) avideo graphics input device adapted to provide color video graphicsinput signals, H) control electronics adapted to: 1) determine the viewdirection of the eye based on the infrared reflection signal, 2)modulate the first set of three visible light lasers based on the videographics input signals and control the first scanner unit based on theinfrared reflection signal to produce, with the scanned foveal laserbeam, color images on the foveal region of the eye, and 3) modulate thesecond set of three visible light lasers based on the video graphicsinput signals and control the second scanner unit based on the infraredreflection signal to produce, with the retinal color and infrared laserbeam: a) color images on a region of the fovea corresponding to a fieldof view of at least 30 degrees×30 degrees and b) infrared reflectedlight for determining the eye view direction.
 2. The display system asin claim 1 wherein the curved mirror is spherical.
 3. The display systemas in claim 1 wherein the curved mirror is ellipsoidal.
 4. The displaysystem as in claim 1 wherein each of the first and second sets of atleast three visible light lasers comprise red, green and blue lasers. 5.The display system as in claim 1 wherein each of the first and secondsets of at least three visible light lasers is a set made up of a red, agreen and a blue laser.
 6. The display system as in claim 1 wherein eachof the first scanner unit and the second scanner unit is comprised of aMEMS scanner.
 7. The display system as in claim 6 wherein the each ofthe first and second MEMS scanner includes a scanner axis that isoperated in a resonant mode.
 8. The display system as in claim 6 whereina horizontal scan for each of the first and second MEMS scanners isprovided by the resonant scanner and vertical scans are provided by aramping voltage applied with respect to one axis of the scanner.
 9. Thedisplay system as in claim 1 wherein the foveal region corresponds to anapproximately 10 degree diameter field of view encompassing the fovea.10. The display system as in claim 1 wherein the second scanner unit isadapted to provide a reflection on the retina corresponding to a fieldof view of about 50 degrees×70 degrees.
 11. The display system as inclaim 1 wherein the second scanner unit is adapted to provide areflection on the retina corresponding to a field of view of having onedimension as large as 120 degrees.
 12. The display system as in claim 1wherein said at least one retina display unit is two retina displayunits and said at least one eye is both of the wearer's two eyes. 13.The display system as in claim 12 wherein said display further comprisesfocus adjuster elements.
 14. The display system as in claim 13 whereinthe focus adjuster elements comprise a variable focus lens and feedbackelectronics adapted to adjust focus of the variable focus lens tomaximize reflection of infrared light detected by said infrared detectorof each of the two retinal display units.
 15. The display system as inclaim 13 wherein the focus adjuster elements comprise a variable focuslens and feedback electronics adapted to adjust focus of the variablefocus lens and said control electronics are adapted to determine thefocus of each of the two eyes by estimating the convergence angle of thetwo eyes.
 16. The display system as in claim 12 wherein the system isadapted to provide three dimensional viewing.
 17. The display system asin claim 16 wherein the system includes a wireless connection to acommunication consol.
 18. The display system as in claim 17 wherein theconsol is a television consol.
 19. The display system as in claim 17wherein the consol is a computer consol in communication with theInternet.
 20. The display system as in claim 17 wherein said system isadapted for computer gaming.
 21. The display system as in claim 1wherein the system is adapted for operation in a virtual reality mode.22. The display system as in claim 1 wherein the system is adapted foroperation in an augmented reality mode.
 23. The display as in claim 1wherein the curved mirror has a varying radius of curvature
 24. Thedisplay system as in claim 1 wherein the system is adapted forimplementation in the form of goggles.
 25. The display system as inclaim 1 wherein the system is adapted for implementation in the form ofa head mounted visor.
 26. The display system as in claim 1 wherein thesystem is adapted for implementation in a form wherein the curvedreflector is a portion of a cockpit window
 27. The display system as inclaim 1 wherein the system is adapted for implementation in a formwherein the curved reflector is a portion of a motor vehicle window